feat(cli): report crashes in the CLI's own code to Checkly [ask] - #1590

Thanks, this is careful work. I've marked which findings were reproduced; the rest come from reading the code.

On the open question: let's drop @sentry/core. #1591 is stacked on this branch and replaces it with a local module of about 120 lines: stack parser, DSN → envelope URL, envelope serialization and crypto.randomUUID(). For the same input it produces the same frames and envelope as @sentry/core 11, apart from a few small differences the PR lists. Merge it into this branch to pick it up.

The size matters beyond the CLI install. A project's package.json usually lists checkly, so the bundled package.json of a Playwright Check Suite does too (bundle.packages.prune is opt-in). Our runners would install and cache @sentry/* (~14 MB) for those bundles, though the CLI never runs there.

Major

1. Errors thrown by built-in methods are never reported (classify.ts, throwingFile). When a built-in throws, V8's innermost frame is <anonymous>, e.g. at String.repeat (<anonymous>). throwingFile returns that, isOwnPath('<anonymous>', …) is false, and the error is classified as not the CLI's. This was reproduced with @sentry/core 11 on Node 24 for 'x'.repeat(-1), [].reduce(fn), new Date('nope').toISOString() and [1].map(undefined). JSON.stringify on a circular object behaves the same way. Suggested fix: skip frames whose filename is not an absolute path or file:// URL (<anonymous>, native, index N) as well as node: frames, so the first real frame decides. Add specs for errors thrown by built-ins.

2. Errors while processing test results are not reported (abstract-check-runner.ts, configureResultListener). this.queue.add(() => this.processMessage(...)) is never awaited or caught, and the reporters' onCheckStart/onCheckEnd run inside processMessage. A TypeError there becomes an unhandled rejection, and Node ends the process before runner.run()'s catch or BaseCommand.finally runs. Nothing is noted or sent. A synchronous throw in the 'message' handler becomes an uncaught exception, with the same result. This is probably where most real bugs in test/trigger/pw-test would occur: reporters and result formatting. The description lists unhandled rejections as a follow-up, but I'd handle this case in this PR. Catch the queued promise's rejection, call noteCliError, and fail the run so the error goes through run()'s catch and the awaited finally.

3. Nothing tells users about the new outbound request. Only the README covers it. There is nothing in the docs site and no release note. Users with egress allowlists or data-handling reviews look there, not at the GitHub README. Please add a short docs page covering error reporting and the two opt-out variables, and mention it in the release notes of the version that ships it.

Do reports get sent before the process exits?

On the paths the PR hooks into, yes. oclif's Command._run awaits this.catch() and then this.finally() before the error reaches handle(), which prints and calls process.exit. BaseCommand.finally awaits reportNotedCliError, so the POST completes, or times out after 3 s, before the exit. This holds for errors that are thrown, this.exit()/this.error(), --json mode, and commands that only set process.exitCode. Nothing in src calls process.exit directly, apart from the hand-off to the project's own CLI. The paths that do lose the report are item 2 above and crashes outside a command's run: oclif config loading, hooks, and bin/launch.cjs.

Minor

• Scrubbing:
  • replaceDir's lookahead does not accept ., !, ] or }. A bare cwd before sentence punctuation (no config in /work/acme-proj.) is left in place, and the home pass misses it when the project is outside home (e.g. cwd=/app/<name>, HOME=/root).
  • The URL-credential regex keeps part of a password that contains @ (http://bob:p@ss@proxy → http://<redacted>@ss@proxy), and key=value redaction stops at a space inside quotes (password: "hunter two" → "<redacted> two"). Rather than patching the regexes, rebuild credential-bearing URLs from scheme and host, and redact quoted values up to the closing quote.
  • Absolute paths outside cwd and home are sent as they are, e.g. a CI checkout under /builds/<org>/<repo> run from a subdirectory with HOME=/root, or --config pointing elsewhere. Consider replacing every absolute path outside the package root.
• No test covers the production wiring. No test covers BaseCommand.catch/finally, longError or the runner noting errors that then get sent. Dropping one of the noteCliError calls, or no longer awaiting the report, would keep every spec green. The DSN can only come from PACKAGE_ROOT/error-reporting.json, so such a test can't be written yet. A small test-only override would allow one.
• Nothing checks that the packed tarball contains error-reporting.json. A packaging mistake would silently turn reporting off. One tar -tzf check in the release job would catch it.
• reportCliError is exported and documented as an entry point, but only the spec calls it.
• Stale paths: the comments in release.yml and .gitignore still name src/services/error-reporting.ts, which is now a folder.
• test-sessions/list.ts now calls noteCliError for parseUnixTimestamp failures. If that ever throws a TypeError/RangeError, a user's bad --from value would be reported as a CLI bug.
• The wait before the error is printed is up to 3 s. That only happens on a real bug, but 1–1.5 s would be plenty for a single POST.
• Two conventions for printed errors: longError(title, err) notes automatically, but longError(title, err.message) and this.error(err.message) need a manual noteCliError(err). Passing the error object everywhere would mean nobody has to remember the manual call.

🤖 Generated with Claude Code